| 1 |
Rethinking Automatic Evaluation in Sentence Simplification
|
|
|
|
In: https://hal.inria.fr/hal-03199901 ; 2021 (2021)
|
|
Abstract:
Automatic evaluation remains an open research question in Natural Language Generation. In the context of Sentence Simplification, this is particularly challenging: the task requires by nature to replace complex words with simpler ones that shares the same meaning. This limits the effectiveness of n-gram based metrics like BLEU. Going hand in hand with the recent advances in NLG, new metrics have been proposed, such as BERTScore for Machine Translation. In summarization, the QuestEval metric proposes to automatically compare two texts by questioning them. In this paper, we first propose a simple modification of QuestEval allowing it to tackle Sentence Simplification. We then extensively evaluate the correlations w.r.t. human judgement for several metrics including the recent BERTScore and QuestEval, and show that the latter obtain state-of-the-art correlations, outperforming standard metrics like BLEU and SARI. More importantly, we also show that a large part of the correlations are actually spurious for all the metrics. To investigate this phenomenon further, we release a new corpus of evaluated simplifications, this time not generated by systems but instead, written by humans. This allows us to remove the spurious correlations and draw very different conclusions from the original ones, resulting in a better understanding of these metrics. In particular, we raise concerns about very low correlations for most of traditional metrics. Our results show that the only significant measure of the Meaning Preservation is our adaptation of QuestEval.
|
|
Keyword:
[INFO.INFO-CL]Computer Science [cs]/Computation and Language [cs.CL]
|
|
URL: https://hal.inria.fr/hal-03199901
|
|
BASE
|
|
Hide details
|
|
|
| 2 |
Multilingual Unsupervised Sentence Simplification
|
|
|
|
In: https://hal.inria.fr/hal-03109299 ; 2021 (2021)
|
|
BASE
|
|
Show details
|
|
|
| 3 |
Automatic sentence simplification using controllable and unsupervised methods ; Simplification automatique de phrases à l'aide de méthodes contrôlables et non supervisées
|
|
|
|
In: https://tel.archives-ouvertes.fr/tel-03543971 ; Computation and Language [cs.CL]. Sorbonne Université, 2021. English. ⟨NNT : 2021SORUS265⟩ (2021)
|
|
BASE
|
|
Show details
|
|
|
| 4 |
Controllable Sentence Simplification
|
|
|
|
In: LREC 2020 - 12th Language Resources and Evaluation Conference ; https://hal.inria.fr/hal-02678214 ; LREC 2020 - 12th Language Resources and Evaluation Conference, May 2020, Marseille, France ; http://www.lrec-conf.org/proceedings/lrec2020/index.html (2020)
|
|
BASE
|
|
Show details
|
|
|
| 5 |
CamemBERT: a Tasty French Language Model
|
|
|
|
In: ACL 2020 - 58th Annual Meeting of the Association for Computational Linguistics ; https://hal.inria.fr/hal-02889805 ; ACL 2020 - 58th Annual Meeting of the Association for Computational Linguistics, Jul 2020, Seattle / Virtual, United States. ⟨10.18653/v1/2020.acl-main.645⟩ (2020)
|
|
BASE
|
|
Show details
|
|
|
| 6 |
ASSET: A Dataset for Tuning and Evaluation of Sentence Simplification Models with Multiple Rewriting Transformations
|
|
|
|
In: ACL 2020 - 58th Annual Meeting of the Association for Computational Linguistics ; https://hal.inria.fr/hal-02889823 ; ACL 2020 - 58th Annual Meeting of the Association for Computational Linguistics, Jul 2020, Seattle / Virtual, United States (2020)
|
|
BASE
|
|
Show details
|
|
|
| 7 |
huggingface/transformers: Trainer, TFTrainer, Multilingual BART, Encoder-decoder improvements, Generation Pipeline ...
|
|
|
|
BASE
|
|
Show details
|
|
|
| 8 |
MUSS: Multilingual Unsupervised Sentence Simplification by Mining Paraphrases ...
|
|
|
|
BASE
|
|
Show details
|
|
|
| 9 |
ASSET: A Dataset for Tuning and Evaluation of Sentence Simplification Models with Multiple Rewriting Transformations ...
|
|
|
|
BASE
|
|
Show details
|
|
|
| 10 |
ASSET: A dataset for tuning and evaluation of sentence simplification models with multiple rewriting transformations
|
|
|
|
BASE
|
|
Show details
|
|
|
| 11 |
Controllable Sentence Simplification
|
|
|
|
In: https://hal.inria.fr/hal-02445874 ; 2019 (2019)
|
|
BASE
|
|
Show details
|
|
|
| 12 |
CamemBERT: a Tasty French Language Model
|
|
|
|
In: https://hal.inria.fr/hal-02445946 ; 2019 (2019)
|
|
BASE
|
|
Show details
|
|
|
| 14 |
Reference-less Quality Estimation of Text Simplification Systems
|
|
|
|
In: 1st Workshop on Automatic Text Adaptation (ATA) ; https://hal.inria.fr/hal-01959054 ; 1st Workshop on Automatic Text Adaptation (ATA), Nov 2018, Tilburg, Netherlands ; https://www.ida.liu.se/~evere22/ATA-18/ (2018)
|
|
BASE
|
|
Show details
|
|
|
| 15 |
ELMoLex: Connecting ELMo and Lexicon features for Dependency Parsing
|
|
|
|
In: CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies ; https://hal.inria.fr/hal-01959045 ; CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies, Oct 2018, Brussels, Belgium. ⟨10.18653/v1/K18-2023⟩ (2018)
|
|
BASE
|
|
Show details
|
|
|
|
|
